SPACE: Structured Compression and Sharing of Representational Space for Continual Learning
نویسندگان
چکیده
Humans learn incrementally from sequential experiences throughout their lives, which has proven hard to emulate in artificial neural networks. Incrementally learning tasks causes networks overwrite relevant information learned about older tasks, resulting ‘Catastrophic Forgetting’. Efforts overcome this phenomenon often utilize resources poorly, for instance, by growing the network architecture or needing save parametric importance scores, violate data privacy between tasks. To tackle this, we propose SPACE, an algorithm that enables a continually and efficiently partitioning learnt space into Core space, serves as condensed knowledge base over previously Residual is akin scratch current task. After each task, analyzed redundancy, both within itself with space. A minimal number of extra dimensions required explain task are added remaining freed up next We evaluate our on P-MNIST, CIFAR sequence 8 different datasets, achieve comparable accuracy state-of-the-art methods while overcoming catastrophic forgetting. Additionally, well suited practical use. The analyzes all layers one shot, ensuring scalability deeper Moreover, analysis translates filter-level sparsity, structured nature gives us 5x improvement energy efficiency during inference state-of-the-art.
منابع مشابه
Understanding failures of learning: Hebbian learning, competition for representational space, and some preliminary experimental data.
Understanding Failures of Learning: Hebbian Learning, Competition for Representational Space, and some Preliminary Experimental Data James L. McClelland1 2, Adam Thomas1 3, Bruce D. McCandliss1 4 and Julie A. Fiez1 4 Center for the Neural Basis of Cognition1, Department of Psychology, Carnegie Mellon University2, Department of Neuroscience, University of Pittsburgh3, Learning Research and Devel...
متن کاملRepresentational neglect and navigation in real space.
Topographical orientation relies on several cognitive strategies adopted by humans to move within the environment. In the present study, we investigate whether mental representation disorders affect specific cognitive mechanisms subserving human orientation. In order to differentiate distinct cognitive mechanisms involved in topographical orientation, we created a human version of the well-know...
متن کاملLearning to Adapt Structured Output Space for Semantic Segmentation
Convolutional neural network-based approaches for semantic segmentation rely on supervision with pixel-level ground truth, but may not generalize well to unseen image domains. As the labeling process is tedious and labor intensive, developing algorithms that can adapt source ground truth labels to the target domain is of great interest. In this paper, we propose an adversarial learning method f...
متن کاملHybrid Teaching-Learning-Based Optimization and Harmony Search for Optimum Design of Space Trusses
The Teaching-Learning-Based Optimization (TLBO) algorithm is a new meta-heuristic algorithm which recently received more attention in various fields of science. The TLBO algorithm divided into two phases: Teacher phase and student phase; In the first phase a teacher tries to teach the student to improve the class level, then in the second phase, students increase their level by interacting amon...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: IEEE Access
سال: 2021
ISSN: ['2169-3536']
DOI: https://doi.org/10.1109/access.2021.3126027